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Introduction 


This  interim  report  constitutes  a  summary  of  research  performed  under 
Grant  AF0SR-81-0047  during  the  year  beginning  October  1,  1982.  First  we 
present  a  list  of  the  personnel  Involved  in  the  research  effort.  In  the  next 
section  we  present  a  summary  of  the  research  results  that  have  been  achieved. 
Then  in  the  following  section  we  briefly  comment  upon  the  research  in 
progress.  This  Is  followed  by  a  list  of  publications  supported  during  this 
grant  year. 
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*Y1h-Chiao  Liu  received  his  M.S.E.  degree  in  May  1983.  His  thesis  was 
entitled  "Some  Relative  Efficiency  Results  for  Detectors  for  Time  Varying 
Signals."  He  is  presently  working  on  the  Ph.D.  degree  in  the  area  of 
signal  detection. 
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Summary  of  Results 

In  this  section  we  will  present  a  brief  summary  of  our  research  results 
published  after  October  1,  1982.  The  reference  numbers  in  this  section  are 
keyed  to  the  publication  list  in  the  last  section.  Most  of  our  research 
results  are  In  the  areas  of  quantization  theory  and  detection  theory.  We 
will  begin  our  summary  by  discussing  our  results  in  quantization  theory; 
this  will  be  followed  by  a  presentation  of  our  results  in  detection  theory; 
and  finally,  we  will  mention  our  results  in  other  areas. 

Quantization  is  the  process  by  which  data  is  reduced  to  a  simpler,  more 
coarse  representation  which  is  more  compatible  with  digital  processing. 

Loosely  speaking,  quantization  is  at  the  heart  of  analog  to  digital  conversion. 
It  is  an  area  which  has  increased  in  Importance  in  the  last  few  years  due  to 
the  burgeoning  advances  in  digital  technology.  The  typical  goal  of  quantiza¬ 
tion  is  to  reduce  data  to  a  simpler  representation  without  causing  much 
distortion;  that  is,  the  output  of  a  quantizer  should  be  close  to  the  input, 

with  some  appropriate  measure  of  distance.  An  N-level  k-dimensional  vector 

k  k 

quantizer  Is  a  mapping  Q:  R  -►R  which  assigns  the  Input  vector  x  to  an 

I# 

output  vector  Q(x)  chosen  from  a  set  of  N  vectors  {y^ :  y^eR  ,  1=1, 

Generally,  the  quantizer  input  is  modelled  as  a  random  vector  X  described  by 
a  k-varlate  distribution  F.  A  measure  of  quantizer  performance  Is  the 
distortion  function 

D(Q,F)  =  Jd(x,Q(x))dF(x),  (1) 

k  k 

where  d:  R  *R  -►R  Is  an  appropriately  chosen  cost  function.  An  optimal 
N-level  quantizer  Q  for  the  random  vector  X  is  one  that  minimizes  (1)  over 
the  class  of  all  N-level  quantizers. 

There  had  apparently  been  a  long-standing  belief  among  researchers  in 
quantization  theory  that  optimal  quantizers  always  exist.  This  existence  Is 
Important  from  the  viewpoint  of  numerical  design  algorithms  and  in  studying 
convergence  properties  of  sequences  of  quantizers;  also,  several  results  in 
quantization  theory  are  hypothesized  upon  the  existence  of  optimal  quantizers. 
Recently  we  have  been  Investigating  this  existence  belief  and  we  have 
produced  several  results  which  validate  It  In  many  cases.  In  [3]  we  showed 
that  If  d(x,y)  =  C(||  x-y J |  ),  where  |j*  ||  represents  the  Euclidean  norm  and 
C:  [0,<*>)  -*■ [0,»)  Is  non-decreasing  and  lower  semi -continuous,  then  an  optimal 
N-level  quantizer  exists  for  any  positive  Integer  N  and  for  any  distribution 


F.  Then  in  [1]  and  [16]  we  considered  non-difference  based  cost  functions, 
e.g. ,  d(x,y),  and  In  a  fairly  general  context  we  established  existence  of 
optimal  quantizers.  Finally,  In  [6]  we  established  conditions  for  optimal 
quantizers  to  exist  in  a  metric  space.  Our  work  on  quantization  of  metric 
spaces  is  the  most  general  treatment  of  quantizers  in  metric  spaces  of  which 
we  are  aware. 

We  have  also  been  active  in  establishing  convergence  properties  of 
sequences  of  quantizers.  These  convergence  results  are  important  from  the 
viewpoint  of  numerical  design  algorithms,  and  they  yield  considerable 
insight  into  the  limiting  behavior  of  sequences  of  quantizers.  Suppose  that 
a  sequence  of  probability  measures  un  converges  weakly  to  a  probability 
measure  p.  Let  Qn  be  an  optimal  N-level  quantizer  for  pn.  Does  the 
distortion  associated  with  the  quantizer  Qn  and  the  measure  pn  converge  to 
the  optimal  distortion  for  quantizing  p  with  N-level s?  Does  the  sequence 
of  optimal  quantizers  for  the  Pn's  converge  to  a  quantizer  Q;  and  if  so,  is 
Q  optimal  for  p?  Several  of  our  convergence  results  have  been  focused  about 
these  two  questions.  In  [12]  we  considered  difference  based  cost  functions, 
e.g.  d(x-y),  and  we  established  results  sufficient  for  affirmative  answers  to 
the  above  questions.  Then  in  [1]  and  [16]  we  established  conditions 
sufficient  for  affirmative  answers  for  non-difference  based  cost  functions, 
e.g.  d(x,y).  In  [6]  we  extended  these  results  to  metric  spaces.  In  each  of 
the  above  works  (i.e.  [12]  for  difference  based  cost  functions," [1]  and  [16] 
for  non-difference  based  cost  functions,  and  [6]  for  metric  spaces),  we  also 
considered  the  above  two  questions  where  pn  represented  the  empirical  measure 
based  on  n  lid  samples  drawn  from  the  measure  p,  and  we  established  conditions 
sufficient  for  almost  sure  convergence  in  the  above  two  questions.  In  all  of 
the  above  convergence  results  we  chose  to  put  conditions  on  the  cost  function 
rather  than  the  distribution;  the  cost  function  is  easier  to  control  than  the 
(frequently  not  exactly  known)  underlying  distribution. 

These  convergence  results  for  sequences  of  quantizers  are  fairly  general 
and  they  form  powerful  tools  for  the  study  of  quantization.  For  example,  one 
of  the  more  practical  problems  associated  with  quantizers  is  the  problem  of 
how  to  construct  them.  Host  of  the  algorithms  for  quantizer  design  Involve 
successively  Improving  a  suboptlmal  quantizer,  with  the  procedure  hopefully 
converging  to  an  optimal  quantizer.  One  of  the  currently  most  popular  design 
algorithms  for  vector  quantization  is  the  so-called  "Linde-Buzo-Gray  algorithm" 
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(IEEE  Transactions  on  Communications,  January  1980).  As  a  by-product  of  one 
of  our  results  in  [12],  we  established  convergence  of  this  algorithm  for  r-th 
power  distortion  measures,  i.e.  d(x,y)  =  ||x-y||r.  This  is  the  first  rigorous 
convergence  result  for  this  algorithm  in  a  reasonably  general  context. 

We  have  also  investigated  some  practical  aspects  of  scalar  quantization. 

A  popular  way  of  realizing  a  scalar  quantizer  is  via  a  method  known  as 
companding.  A  companding  system  consists  of  an  invertible  function  G:  F 
[0,1]  followed  by  a  uniform  N-level  quantizer  on  [0,1],  followed  by  the 
inverse  function  Any  arbitrary  N  level  scalar  quantizer  can  be 

realized  via  a  companding  system.  (For  k>l,  this  statement  is  false  in  1R  .) 
In  [4]  we  considered  the  situation  where  a  noisy  channel  was  inserted  between 
the  uniform  quantizer  and  the  inverse  function  G’^(*).  Thus  the  input  X  goes 
through  the  "compressor"  G,  the  output  is  then  uniformly  quantized  and 
transmitted  over  a  noisy  channel,  and  the  result  is  put  through  the  "expander" 
G'1.  Ideally,  the  output  should  be  close  to  the  input.  In  [4]  we  assumed 
that  the  number  of  quantization  levels  N  was  very  large  and  we  gave  an 
approximation  for  the  compressor  G  that  took  into  account  not  only  the  error 
due  to  the  quantization  but  also  the  error  due  to  the  channel  noise. 

In  [15]  we  presented  a  simple  and  straightforward  technique  for  construct¬ 
ing  minimum  mean  squared  error  symmetric  uniform  scalar  quantizers  for  some 
common  distributions  on  the  data. 

In  the  context  of  scalar  minimum  mean  squared  error  quantization,  one  of 
the  most  popular  design  techniques  Is  the  Lloyd-Max  algorithm  (IRE  Transactions 
on  Information  Theory,  March  1960  and  IEEE  Transactions  on  Information  Theory, 
March  1982).  Unfortunately,  two  potential  problems  arise  with  the  Lloyd-Max 
scheme.  The  first  problem  Is  how  to  get  a  good  initial  guess  for  starting  the 
iterative  scheme,  and  the  second  problem  Is  how  to  Intelligently  update  the 
algorithm.  Both  of  these  problems  were  addressed  in  [10]  for  some  common 
distributions  on  the  data.  Our  modifications  of  the  Lloyd-Max  algorithm 
result  In  a  very  fast  design  algorithm  for  scalar  minimum  mean  squared  error 
quantization.  For  example,  we  can  design  a  64-level  quantizer  for  a  Gaussian 
distribution  with  a  high  degree  of  accuracy  (the  terminating  condition  for 
the  Lloyd-Max  algorithm  was  set  at  10”8)  In  0.136  seconds  of  computer  time  on 
a  CDC  Cyber  170/750. 

Another  research  area  In  which  we  have  recently  obtained  results  Is  the 
area  of  signal  detection.  The  detection  problem  Is  modeled  as  a  test  between 
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two  statistical  hypotheses;  we  assume  that  under  the  null  hypothesis  noise 
alone  Is  being  observed,  and  under  the  alternate  hypothesis  a  signal  plus 
noise  is  being  observed.  In  discrete  time  detection  we  assume  that  the 
observation  is  Indexed  by  a  subset  of  the  Integers,  e.g.  x1,x2,...,xn,  and 
in  continuous  time  detection  we  assume  that  the  observation  is  Indexed  by 
an  Interval,  e.g.  x(t),  0<_t<J.  In  [9]  we  considered  using  a  continuous 
time  filter  for  a  discrete  time  problem.  That  Is,  we  supposed  that  the 
signal  s(*)  was  known  at  certain  discrete  Instants  of  time  {t^},  and  we 
showed  that  under  certain  regularity  conditions  (including  bandlimltedness 
of  s(t))  we  could  derive  a  signal  §{•)  such  that  s(tk)  =  s (tk)  in  such  a  way 
that  if  we  used  a  continuous  time  filter  based  on  the  signal  s(t),  0<.t<J, 
then  the  performance  of  the  detector  was  robust  to  the  Inexact  knowledge  of 
the  actual  signal  s(t),  0<t<J.  We  showed  that  the  performance  of  the 
continuous  time  detector  based  on  the  signal  s(t),  0<t<T,  could  be  consider¬ 
ably  better  than  the  performance  of  the  discrete  time  detector  based  on  the 
signal  (s(tk) } ,  even  though  the  signal  s(t).  Is  not  known  for  t  outside  the 
set  {tk>. 

In  the  case  of  discrete  time  detection  where  the  noise  and  the  signal 
are  stationary  and  the  samples  are  Independent,  It  is  well  known  that  the 
Neyman- Pearson  test  has  a  test  statistic  which  can  be  expressed  as 

l  8(x<> 

where  1*1,. ...n,  represent  the  observations,  and  g(*)  is  an  appropriately 
chosen  function.  In  earlier  work  we  had  considered  the  problem  of  constrain¬ 
ing  the  test  statistic  to  be  of  the  above  form  and  letting  the  noise  samples 
be  "slightly"  dependent.  We  then  tried  to  choose  the  function  g(*)  to  best 
account  for  the  dependency  structure  of  the  noise,  in  the  sense  of  the 
asymptotic  relative  efficiency  (or  Pitman  efficiency)  with  respect  to  any 
other  choice  for  g(>).  In  [14]  we  Investigated  the  problem  of  how  to  choose 
g(*)  when  both  the  signal  and  the  noise  were  modelled  as  ^-mixing  random 
processes,  where  we  also  allowed  the  noise  to  be  dependent  on  the  signal  over 
a  finite  window,  such  as  signal  dependent  noise  Induced  through  reverberation 
effects.  In  [13]  we  considered  the  problem  of  approximating  an  optimal  g(>) 
by  a  sequence  of  Borel  measurable  functions  (g^ (• ))•  We  compared  the 
performance  resulting  from  the  approximate  nonl Inearl  ties  to  the  optimal 
performance,  and  we  showed  that  the  loss  In  performance  can  be  made  arbitrarily 


small  by  making  g^*)  appropriately  close  to  g( • ) .  We  allowed  a  strong-mixing 
dependency  structure  for  the  (random)  signal  and  the  noise,  and  we  considered 
as  examples  specific  forms,  e.g.  quantizers,  polynomials,  for  the  g^(*)« 

In  [11]  we  considered  the  discrete  time  detection  of  time  varying 
deterministic  signals  in  weakly  dependent  noise.  We  considered  a  finite  set 
of  observations,  and  we  were  concerned  with  the  situation  where  the  dependency 
in  the  noise  was  Ignored;  that  is,  the  detector  was  designed  assuming  that 
the  noise  samples  were  mutually  Independent.  We  investigated  quantitative 
conditions  which  allowed  determining  when  the  dependency  can  be  ignored,  and 
we  presented  a  result  which  allowed  bounding  the  variation  in  false  alarm  rate 
and  detection  probability  induced  by  Ignoring  the  dependency. 

There  has  recently  been  a  great  deal  of  Interest  in  robust  hypothesis 
testing,  and  engineers  have  attempted  to  apply  results  in  this  area  to 
problems  in  signal  detection.  This  may  be  attractive  when  one  realizes  that 
the  statistical  characterizations  adopted  by  engineers  are  frequently  only 
reasonable  approximations.  In  [5]  we  studied  the  actual  performance 
associated  with  a  robust  test  for  a  constant  signal  In  i id  Laplace  noise,  and 
we  showed  that  in  many  cases  the  degradation  In  performance  (l.e.  the 
performance  loss  due  to  the  robust  test  as  compared  to  the  performance  of  the 
Neyman-Pearson  optimal  test)  was  Indeed  very  small. 

The  relative  efficiency  between  two  detectors  is  a  ratio  of  the  amount 
of  data  required  by  one  detector,  relative  to  another,  to  attain  a  prescribed 
level  of  performance.  Although  this  concept  Is  of  fundamental  Importance  in 
the  theory  of  signal  detection.  It  has  been  successfully  investigated  In  only 
very  few  special  cases.  As  an  approximation  to  the  relative  efficiency, 
engineers  have  frequently  employed  the  asymptotic  relative  efficiency  (ARE), 
the  limiting  value  of  the  relative  efficiency  (under  suitable  regularity 
conditions)  as  the  sample  sizes  required  by  the  detectors  approach  Infinity. 
The  ARE  was  Introduced  In  the  statistical  literature,  where  It  Is  known  as 
the  Pitman  efficiency.  Usually  It  can  be  determined  In  a  fairly  straight¬ 
forward  fashion,  and  this  Is  due  principally  to  an  appeal  to  the  central 
limit  theorem.  The  ARE  Is  a  limiting  result;  and  In  any  practical  engineering 
situation,  only  a  finite  number  of  samples  can  be  taken  In  the  context  of 
discrete  time  detection.  Thus  It  might  not  always  be  appropriate  to 
approximate  the  relative  efficiency  with  the  ARE.  In  [7]  we  considered  the 
discrete  time  detection  of  a  known  time  varying  signal  In  additive  noise. 
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where  the  noise  sequence  Is  assumed  to  be  a  sequence  of  lid  random  variables; 
and  we  studied  the  relative  efficiency  of  the  sign  detector,  a  popular 
nonparametrlc  detector,  and  the  correlation  detector,  which  Is  Neyman-Pearson 
optimal  In  the  case  when  the  noise  Is  Gaussian.  In  this  work  [7]  we  presented 
results  illustrating  the  convergence  of  relative  efficiencies  for  both 
Gaussian  noise  and  Laplace  noise.  Some  examples  were  given  where  the  relative 
efficiencies  did  not  quickly  converge  to  the  ARE. 

In  another  area  of  research,  we  have  investigated  some  effects  of  a  form 
of  nonlinear  distortion  on  spectral  properties  of  random  processes.  Let 
{X(t),teF}  denote  a  second  order  random  process  that  is  second  order 
stationary  (I.e.  the  bivariate  distribution  is  invariant  to  a  translation  of 
the  time  axis),  and  mean  square  continuous.  Then  it  Is  well  known  that  X(t) 
possesses  a  spectral  distribution  function  F.  (We  take  all  spectral  distri¬ 
bution  functions  to  be  zero  at  minus  infinity.)  Let  g:  1R  be  a  Bore! 

o 

measurable  function  such  that  E{(g[X(t)])  }<“.  Then  g[X(t)]  Is  mean  square 
continuous  and  thus  possesses  a  spectral  distribution  function  G.  For  p>2, 
let 

f|u|pdF(u) 

A  z  ■ 

P  F(») 

and  let 

J|u|pdG(u) 

*P  "  GW  ’ 

where  we  assume  that  F(»)  >0  and  G(«)  >0  (i.e.  neither  X(t)  nor  g(*)  Is 
"degenerate").  The  and  represent  the  p-th  absolute  moments  of  the 
Input  and  output  normalized  spectral  distributions,  respectively.  These 
quantities  are  Important  since  they  relate  to  measures  of  spectral 
dispersion,  and  thus  bandwidth.  Bandwidth  is  a  fundamental  property  of 
signal  transmission;  and  nonl Inearl ties,  such  as  g(»),  are  encountered  in 
numerous  situations.  In  [2]  we  compared  Ap  to  ^  for  various  conditions 
placed  on  g(«)  and  X(t),  and  with  certain  conditions  on  X(t)  we  gave  both 
upper  and  lower  bounds  to  In  terms  of  Ap  and  g(*).  For  example.  It 
follows  from  [2]  that  If  X(t)  Is  a  zero  mean  Gaussian  process  with  a 
nonconstant  autocorrelation  function  and  If  g(>)  Is  odd  and  not  almost 
everywhere  constant,  then  Aj  <  ^»  with  equality  If  and  only  if  g  Is  almost 
everywhere  linear. 
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In  [17]  we  Investigated  an  existing  method  (Delp  and  Mitchell,  IEEE 
Transactions  on  Communications,  September  1979)  for  image  compression  known 
as  block  truncation  coding.  The  basic  block  truncation  coding  approach 
employs  a  two  level  quantizer  whose  output  levels  are  obtained  through 
matching  the  first  two  sample  moments  of  the  data  before  and  after  quantiza¬ 
tion.  We  generalized  this  basic  block  truncation  coding  approach  by  using 
two  level  quantizers  which  preserve  higher  order  moments.  This  generaliza¬ 
tion  offered  the  potential  for  improved  performance.  Some  examples  were 
given  to  illustrate  the  improvement  in  image  quality. 

To  conclude  the  survey  of  our  recent  work,  in  [8]  we  summarized  several 
mistakes  that  are  frequently  made  in  communication  theory.  For  example,  if 
X(t)  is  a  zero  mean  stationary  Gaussian  process  and  R(t)  *  E{X(t)X(t+t)}, 
then  it  is  possible  for  X{t)  to  be  ergodic  but  R(t)  to  not  converge  to  zero 
as  t -*•».  We  point  out  that  ergodicity  does  not  imply  "asymptotic  indepen¬ 
dence".  As  another  example,  we  point  out  that  the  conditional  expectation 
does  not  always  yield  a  minimum  mean  squared  error  estimate.  That  is,  we 
constructed  two  bounded  random  variables  X  and  Y  and  a  function  f:  F^IR 
such  that  Y  * f (X)  pointwise  on  the  underlying  probability  space  but 
E{[Y-E(Y|X}]2}>1010. 


I 
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Research  in  Progress 

Our  research  Is  progressing  very  well  in  several  directions.  Host  of 
our  efforts  are  concentrated  upon  a  rigorous  analysis  of  problems  in  signal 
processing  that  are  dictated  by  practical  concerns,  and  most  of  our  current 
research  Is  directed  toward  non-Gausslan  phenomena.  In  the  remainder  of 
this  section  we  will  briefly  describe  the  problems  we  are  currently 
investigating. 

Ne  are  presently  studying  algorithms  for  quantizer  design,  and  we  are 
considering  both  the  case  where  the  distribution  of  the  data  to  be  quantized 
is  known  and  the  case  where  it  is  unknown.  For  the  case  of  a  known  distribu¬ 
tion,  we  are  currently  approaching  the  Lloyd-Max  algorithm  from  the  viewpoint 
of  a  problem  in  control  theory.  That  is,  one  can  view  a  "run"  through  the 
algorithm  as  a  plant,  and  one  can  view  the  updating  of  the  algorithm  as  a 
feedback  control.  We  can  thus  view  the  fast  convergence  of  the  algorithm  as 
a  matter  of  using  a  good  feedback  In  the  control  system.  We  have  recently 
successfully  modified  the  Lloyd-Max  algorithm  in  the  context  of  scalar 
minimum  mean  squared  error  quantization,  and  we  are  presently  extending  this 
analysis  to  minimum  mean  absolute  error  quantization. 

In  practice  the  distribution  of  data  to  be  quantized  Is  often  unknown 
and  one  must  use  an  estimate  of  the  true  distribution.  Consider  an  algorithm 
for  vector  quantization  which  assumes  a  known  distribution  for  the  Input.  If 
an  estimate  of  this  distribution  were  used  Instead,  what  kind  of  performance 
can  we  expect?  Under  what  conditions  can  we  get  strongly  consistent 
estimates  of  the  optimal  output  levels?  We  are  currently  attempting  to 
answer  these  questions  in  a  reasonably  general  sense. 

Recently  a  group  of  researchers  (Gray,  Kleffer,  and  Linde,  Information 
and  Control,  May  1980)  used  a  sample  empirical  distribution  as  an  estimator 
In  the  so-called  Li nde-Buzo-Gray  algorithm.  However,  their  analysis  holds 
only  for  this  algorithm.  Will  a  similar  result  hold  for  other  algorithms? 

We  are  currently  investigating  this  general  Idea  In  an  "algorithm- free" 
context;  that  Is,  we  are  attempting  to  derive  results  which  are  Independent 
of  the  particular  algorithm  used. 

Another  area  we  are  currently  Investigating  Is  the  relative  efficiencies 
of  detectors.  Consider  two  discrete  time  detectors,  say  Dj  and  D2.  Assume 
that  for  detector  0^,  the  minimum  number  of  samples  needed  to  achieve  a  false 
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alarm  probability  no  greater  than  a  and  a  detection  probability  of  at  least 
0  is  n...  Then  the  relative  efficiency  of  D2  with  respect  to  D1  is  n^/n^. 

In  the  context  of  discrete  time  detection,  the  relative  efficiency,  as  a 
function  of  n^ ,  has  rarely  yielded  to  successful  analysis.  Our  preliminary 
results  in  [7]  are  notable  in  the  sense  that  they  presently  represent  the 
most  successful  approach  to  the  study  of  the  relative  efficiency  in  the 
literature.  We  are  currently  extending  these  results  to  the  case  where  the 
discrete  time  detection  is  in  the  context  of  dependent  noise.  We  plan  to 
also  consider  the  situation  where  the  time  varying  signal  (s^ >  is  not 
precisely  known. 

The  above  summary  describes  our  on-going  research.  In  the  near  future 
we  plan  to  also  consider  some  aspects  of  non-Gaussian  random  processes,  and 
we  also  plan  to  investigate  the  performance  of  detectors  and  how  this  depends 
on  errors  or  inaccuracies  in  the  assumptions  governing  the  design  of  the 
detection  system. 
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